CS 2980 : Model - based Semantic Compression in Database Project Report

We are now in big data era, advances in data collection and management technologies have led to large databases. For example, domains such as Medicine, Biology, Music and experimental sciences in general, are all characterized by large data sequences. Considering the amount of space the big data required and the amount of IO required to fetch the big data, data compression has become an efficient way to reduce the database size and improve the query performance. Data compression can be lossless or lossy. Lossless compression aims to provide the exact query answer, whereas lossy compression aims to minimize the amount of data that needs to be held. Obviously, lossless compression needs more computation and metadata to preserve the entire information. Also, providing an exact answer to a complex query in large data recording environment can take minutes, or even hours, while users are more prone to searching for certain patterns of behavior that they approximately visualize, rather than for specific values. Thus, lossy compression become popular in analytical use. Semantic compression is a kind of lossy compression. Since semantic compression account for and exploit both the meanings and dynamic ranges of error individual attributes; and existing data dependencies and correlations among attributes in the table [3], it is very effective for table-data compression. In this project, columns are compressed based on their associations with other columns by using predictive models. Weka is a comprehensive suite of Java class libraries that implement many machine learning and data mining algorithms. It contains implementations of many algorithms for classification and numeric prediction [7], which can help us to build predictive model for target columns to achieve semantic compression. Inspired by semantic compression and Weka, this project implemented a system that columns are compressed based on their associations with other columns by using Weka models. The system will predict the result rather than read it directly from database when given a SQL query together with an error tolerance, therefore to achieve higher performance improvements in speed.